An Exploration of Approaches for the Stanford Question Answering Dataset

In this paper, we present an exploration of several approaches of varying complexity, novelty and effectiveness applied to solving the reading comprehension problem evaluated by the Stanford Question Answering Dataset (SQuAD), taken from the perspective of a novice practitioner of Deep Neural Networks for Natural Language Processing. Here, we present several models, their mathematical structure, analysis of their implementation and evaluation of their effectiveness in retrieving answers to questions on examples from the SQuAD dataset. In particular, we evaluate (1) a simple sequence attention-mix boundary model, (2) a more complex sequence attention-mix token model and (3) an implementation of a Match-LSTM model for reading comprehension. For each model, we also provide the results of a thorough hyperparameter search and lessons learned from implementation the model in the TensorFlow framework. Our final model achieves an F1 score of 57.7% and an exact match score of 46.3% on the official SQuAD test set.1 We conclude with notes on the training infrastructure we built to effectively support model training and hyperparameter search, and valuable practical lessons learned from the operation of a physical networked computer cluster for neural network training and evaluation.